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I. INTROLUCTION 

Fer the remainder of this decade and beyond, the Navy is 
faced with the diffictlt prceblem of attracting and retaining 
sufficiert rersonnel to meet its ever increasing manpower 
reguirenents. The fleet is expanding toward 600 shirs while 


the availakle manpower pool cf 17 to 21 year old mer and 


women is frojected tc decline. Each year, mililicrs o£ 
dellars are spent recruiting, training and maintainoing 
enlisted Lersonnel. Numerous enlistment standards acdels 


have keer develofred tc imprcve the screening, selecticn and 
asSignment processes for all Navy ratings. Cert levi 
enlistmert standards research is important since it may 
imprcve wanfpower planning, reduce attrition, enhance job 
performance, and lergthen careers. TE) 1s through such 
research that the ultimate gcal of increased readiness at 
lower cost aay be realized. 


Ae EUBECSE OF ANALYSIS 


This study attempts firstly tc improve ufon the methcd- 
clogy presently utilized to develop enlistment standards 
models. In particular, different techniques for develcping 
such godels are gresented, along with alternative criteria 
for measuring successful perfcrmance. A means of measuring 
the utility, or usefulness, of such efforts is also 
Frovided. An attempt has been made to present these methcds 
in a clear manner sc that thcse researchers who fcellcw may 
more readily understard the frocess. The analysis expands 
upon the experience cf numerous similar efforts, including 
several graduate theses prepared at the Naval Postgraduate 


Schocl and many re€ésearch frojects conducted under the 


auspices cf the Navy rersonnel Research and Develcyaenat 
Center (NFRIC) and tke Center for Naval Analyses (CNA). 

The seccndary ptrpose cf this study is te disccver if 
the selecticn standards for cne particular Navy rating fay 
ke imgrcved upon by aralyzing data available at the timze of 
enlistmert. Most predictive npodels developed to date have 
focused cn successful coupletion of technical training 
schocls, cr on attrition. This study is aligned with a tucre 
recert analytic trend of attempting to predict successful 
cperaticral performarce in the ileet. This Aapprcaca wee 
considered most apfyropriate in todayts highly technical 
Navy. The tremendous cost, in terms of both dcliars and 
time, asscciated witk trainirg and retaining Navy perscnnel 
demands gaximum return on investment. By focusing cn cfpera- 
tional success tc develop a larger, more experienced career 
force, tkeéere exists the potential to reduce the burden of 


recruitirg and training new enlistees. 


Fe RATING SELECTED FCR ANALYSIS 


Tc acccaplish the above stated purposes, data fertaining 
to cfeéeraticnal perfcrmance of personnel inthe Aviation 
Machinist's Mate (ATL) ratirg were analyzed. ACs are 
aircraft €ngine mecharics who inspect, adjust, test, lLcepair 
and cverkaul engines used in all Navy airplanes and heiiccp- 
ters. ATs also gferform rccutine maintenance, prepare 


aircraft for flight, and assist in handling aircraft cn the 


ground cr aboard skips. They perform maintenance and 
servicing on either jet or reciprocating engines, ard on 
subsysteas such as fuel, oil, induction compressicn, 


combusticn, turbine ard exhaust. Other AD functiors include 
supervising maintenance, analyzing fuel and oil sanmfles, 
keeping reccrds, evaluatinc engine performance, and g£ain- 


tainitg accessory comgfonents, drive systems and gear [cxes. 
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Aviaticn Machinist's Mates are assigned primarily to 


Naval Aviation Sguadrons cr to Alcterait intermediate 
Mainterance Departmerts. These assignments may te e€itter 
aflcat cr ashore. AIs may also be assigned as instructors 


in training activities, and they are eligible te vclunteer 
mOnmEtligkt duty aS airerewmen, [Ref. 1] 

Fresently, there are over 13,000 men and women assigned 
mor time AL ymating. Since ADS,are assigned to virtually every 
Navy aviation unit, they represent a vital element in 
ensuring a high degree of aircraft readiness iS fairtained. 
As suct, the overall rission effectiveness of Naval Aviaticn 
units is directly linked tc the guality and perfcrmaice of 


memters cf the AL rating. 


C. CERGANKIZATION OF TEIS STUDY 


This chapter has discussed the purpose cf this study, 
and dé€scrikted the AD rating and its importance to the Navy. 
The next chapter will precvide background informaticr on 
enlistrert standards research, and present in general terms 
the eé€vcluticn cf predictor and criterion variables that 
emerged during the development of the models contained in 
this research. Charter ILI describes the data base and AD 
Cata set tkat provided specific measures of oferatioral 
Ferftcrmance for analysis and medel formulation. Chapter IV 
Freserts the three statistical analysis techniques enzployed 
in develcfing six erlistment standards models. lat temay 
Frovides the method and results of the utility analvsis 
conducted on these mcdels. Utility analysis represerts a 
Means Ey which the usefulness of Similar efforts may be 
measured. Chapter VI draws conclusions from the analysis 


and presents reccmmercations for further research. 
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It, SELECTICN OF VARIAELES 
This chapter gites a brief description of some cf the 
selecticn frocedures in use at the time of the data cclléec- 
tion. The results cf previous research on predictirs job 
Ferformance are reviewed and predictor and critericn vari- 


ables tkat have keen shown to ke useful are identified. 


Aw. SFLECTICN BACKGRCUND 


At the time tte data used in this analysis was 
collected, the Navy ccnsidered a number of applicant charac- 
teristics tc guide selecticn and classification decisiors. 
These characteristics included education, high schocl degree 
status, age, number of dependents, scores on the 12 Arsed 
Services Vcecational Aytitude Eattery (ASVAB) subtests, and 
some ccrposite scores. The Armed Forces Qualificaticn Test 
(AFC1) is cre of these composite scores, and an afrlicant's 
score on the AFQI depended on the sum of his sccres cn the 
ASVAE suktests Arithmetic Reascning (AR), Spatial Percefrtion 
(SP), and Werd Knowledge (WK). The AFQT score was reperted 
as a fercentile--a sccre cf &0 meant that an apflicant’s 
total sccre on tke tkree surtests was higher than the scores 
achieved Ey 79 fercert of his peers. The AFQT rercentile 
score was also used tc classify the applicart into cne of 
five mental catesories or AFQT groups. For example, these 
with a score of $0 or better were classified in mental grcup 
I, ard those with a score cf 10 or less were classified as 
grour Vs. 

Arctker composite score is the Success Chances of 
Recruits Entering the Navy (SCREEN) score. This sccre is 


derived frem the personal characteristics of age at entry, 


years of sckooling, whether cr not the applicant had depen- 
dents, and AFQT percentile sccre. This compcesite sccre has 
keen used Ey recruiters since Cctober 1976 in determininfg an 
applicant's eligibility to enlist. [Ref. 2] 

A final composite score that is used in classifying 
Iecruits tc the AD rating is made up of the sum cf the 
Iecruit's standardized sccres on the ASVAB suktests 
Arithmetic Reasoninc (AR), Electronic Informatior (El), 
Generali Science (GS), and Mathematical Knowledge (MK). A 
fininugr sccre of 19C on this ccmposite was necessary for a 


Lecruit to enter the AD rating. 


EF. REVIEW CF PREVIOUS MILITARY STUDIES 


Studies on predicting military job performance have 
Bainly cencentrated cr the survivability of recruits through 
varicus stages cf tkeir military careers. These staces 
include recruit training, Class "A" School, first two years 
cf enlistment and first term of enlistment. 

Lurie used number of dependents, years of education and 
AFQT sccre to predict the perfcrmance of the Electrecric's 
Technician (ETN) ancé Ship's Serviceman (Sd) ratings. He 
found that for the SH rating, non-hign schcol graduates with 
dower AFCI scores rere prcmoted faster than these with 
higher sccres, althcugh AFQGT score had no impact cn first 
term strvival. Tre AFQT score did aid in predicting 
advarcement and survival for members of the EIN rating. 
[Ref. 3] In another study of eight year survival trates, 
Lurie fcund that education level was the most izgpertant 
Predictor. Interestingly he also found that mental grcuf I 
recruits had the worst record in surviving Class "A" Schocl. 
[Ref. 4] | 

A study on the factors affecting first term survival and 


reterticr krehavior cf Machinist's Mates (MM) and Bciler 


Technicians (BT) was conducted by Fietcher in 1975S. He 
found that EIs with creater than ti years of educaticumiag 
greatly improved charces of surviving their first term of 
enlistment. For MMs, those in the iowest and highest rental 
groups had greater survival proktability than others. Foe 
froth ratings, older men had a higher frobability of 
survivai. In relaticn to reenlistment, those BTs with 12 or 
more years cf education had a lcw probability of reenlist- 
ment, sthiie with MMs, having a dependent increased the freb- 
ability cf reenlistmert. [ Ref. 5] 

Studies of military jok -erformance have also investi- 
gated tke effect of the Delayed Entry Program (DEF) on 
survival. Lockman fcund that if frecruit guality (as reas- 
ured ky SCREEN) and training guarantees were contrclled fcr, 
those the were in tke LEP for three or more months hac the 
highest survival rates [Ref. 63. Thomason found that DEP, 
age, e€ducation, recruit training locaticn, Eaceé, numbemeen 
dependents, mental group ard follow on tour assignments had 
varyinc degrees of significance in determining first° tern 
Ssurvivarility [Ref. 7. 

Mcre recent studies have favored the use of gtrltiple, 
rather tran single mgé€asures cf job performance. This is 
recause it is rare tkat a Single measure adéguately ccvers 
the full sccpe cf jck performance. One approach has Leen to 
€xpané tle survivability criteria to include other mgeasures 
c£ jck perfcrmance, such as eligibility to reenlist and the 
achievement of certain paygrades. A continuous criteéericn is 
not availatle under this aprroach; Sailors are either 
categcrized as a success or as a failure. Nesbitt [Ref. 8] 
and Snyder and Bergazzi [Ref. 9] defined various degrees of 
success cr failure in their studies in an effort to generate 


Lore variakility on the criterion. 
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C. CEITERICN ANT PRELTICTOR VAFIABLES 


In most cases sten a sirgle job performaace measure 
{critericn) has been used in previous research, a measure of 
survival has been the cverwhelming choice. This is fecause 
such a criterion is readily measurable, is continucus, and 
is of imgertance to tke Navy since tae costs associated with 
attriticr and <¢ubsecuent replacement are considerable. 
Cther singlé criteria have Leen length of service (10S), 
time te pregoticn, highest rank cr grade achieved, retertion 
{as measured by reerlistment choice), and perfcrmzance at 
erase “A" Schools. 

Tke ccrenaon predictcrs of jcb performance are educaticna, 
Number of devendents, age, sex, race, ASVAB subtest sccreés, 
AFQT scores, mental group, DEP status, and some “after 
@ecessicn" varianples such as recruit training Il1ccatica, 
subsecuent dependent status, and initial and follow cn duty 
assigrmerts. 

In this study txwce criteria will be considered. The 
first still be an LCE criterion andthe second will bea 
compcsite criterion where success will te defined as 
completing the first term of enlistment, being eligikle for 
reenlistment, and achieving the fpaygrade E-4. The candidate 
predictor variables will be age at entry, sex, race, entry 
EFaygrade, education, dependent status, term of enlistmsrent, 
ASVAE suktest scores, AFQT sccres and the composite eccre to 
gualify for the AD rating. The specific variables from the 
AD data set used for analysis, as well as the evoluticn of 


the cata set, are discussed in the next chapter. 


Tit. LATA FASE DEVELOPNENT 
This chapter provides inforration concerning the master 
cata kase and the stuttset of this master file, the AD data 
set, that was used in this study. The generation cf this AD 
data set 1s descrikted in detail, as are the specific 
Fredictcr and critericn variakles discussed in the frevicus 


chapter. 


A- MASTER FILE 


Erlisted anistory records cn over 206,000 ncn-frior 
service sailors who entered the Navy during the fericd 1 
Septegker 1976 to 2:1 December 1978 were ccupiled cy the 
Tefense Manpower Data Center (DMDC) staff. This master file 
was created by mercging data elements frem four sé€farate 
files: (1) the DMDC Cohort file, which 15 i1tselit a com ei. 
tion cf infcrmation from DMDC‘s Enlisted Master Reccre and 
loss files; (2) a Navy Health Research Center (NHEC) file; 
(3) a prcemetion examiration file; and (3%) a Chief cf Naval 
Educaticr and 2eaining (Ns peers ce 

Tre [TRCC Cohort file contains personal and derograrkic 
cata cr incividuals at the tige they entered the service. 
Additionally, it ais updated quarterly by the ~*H#ilieae 
Ferscnnel Ccmmands with active duty or service seraratioa 
infornmaticn as apprcpriate. This file provided cver 150 
variatles te the master file. 

The NHFC file ccntains information on each enlisted 
memker of tke Navy wkc has reen or still iS cn active duty. 
It 1s upcated guarterly from Navy Military Personnel Ccanmand 
{NMPC) change tape extracts, and follows a service nemrer 
from cate cf enlistmert to date of discharge. Tae NERC fiie 


represents approximately 30 variables in the master file. 
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The pregotion exaninaticn file includes advancement exam 
and premcticn data, and the CNET file contains infortration 
cn fcrmal training received by individuals in the data [Lase. 
Togetter these files provided over 60 variables to _ the 
master céta Lase. 

The master file, containing 243 variables, is maintained 
at the Naval Postgractate School. The final update tc the 
file includes DM’C data current as of 30 September 1S&2Z, and 
NHRC data current as of July 1982. The precgram tc access 


data cn tke selected rating is contained in Appendix A. 


EF. AL LATA SET 


Tkis section describes the evolution of the AD data set 
that ccntains the observaticns and measures analyzed in this 
study. The AD data set was derived tnrough a nunmter of 
aterative screens, ard new variabties were created, in crder 
to remcve akerrant olfservations and to refine the predictor 
SmmeGLitcrlicn variables pricr to Statistical analysis. These 
iterative steps ultimately reduced the number of casés in 
the AD data set fren 5,562 to 2,820 observations. The 
Frograms used to screen the data and to create new fredictor 
and critericn variatles are cortained in Appendix A. The 
logic for these processes is discussed in the remainder of 


this charter. 
1. Ecreens 


Since one furfose of this study was tc analyze 


Aviaticn Machinist's Mates whe had operational experience in 


the fleet, the first screen performed on the data was to 
select crly those cases whose final DMDC rate (DMICRATE)1} 
appeared in the last master file update as ADs. This geans 


ane actual variatle nage associated with th2 comment js 
pee + ed in parentheses taroughout the remainder ct this 
conapter. 
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they were «orking ir the AD rating at eitner the tine of 
their separation fren the service, or at the time ci the 
last file update if they were still on active duty. This 
screen allcwed for feople to migrate into the AD trating 
while ensuring that tkose cases who left for anotaér rating 
were excluded frem tke analysis. 

The cases were next screened for ADS with fe fricr 
Navy service (PRIORSEV). In addition, individuals whem 
have charged their sccial security number (SSNCHNGE) were 
removed frca the saaujle. These screens ensured that no 
cbservaticns were ccurted mcre than once in the analysis. 

The observaticns in the AD data were then screened 
to select cnly those people who were tested on ASVAE forms 
5, 6, cr 7 (TESTFORN) vat cadistnent. These test forme were 
in use dtring the feriod in which the individuals in the 
Gata set enlisted. Also, those cases whose subtest ecores 
(ASVAES) were inpossibly high were eliminated from the data 
set. 

Iwc other screens were conducted to capture those 
cbservations whe enlisted in either the Regular Navy or 
Naval Feserve (SERVACCS), and who were known to have signed 
up fcr at deast four years active service (RECENLST). it 1s 
worth reting that dtring acdel develofnent, the tena Won 
enlistment measure (IEFRMENLI) was consistently significant, 
rut sith a negative value for the parameter estimate. Tkis 
indicated that those individuals who enlisted for Milcnger 
cblicated service actually served less time than thcse sho 
signed up fcr shortei terms of enlistment. The parameter 
estinate fcr tern of enlistment changed tc a positive value 
when the RECENLST screen was inplemented. Apparently, by 
screening fer four year active duty obligors, the cata set 
then excluded those reservists who were required tc serve 
cniy three years of their six year obligation cn active 
duty. Fer these observations, a six year term of enlistment 
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waS an errceneous value for the TERMENLT variable. This 
important discovery reveals a probable flaw in earlier 
similar enlistmert standards analytic efforts. 

Ancther screen facilitated inclusion of those cases 
for wkich tltere was clear indication of taeir eligibility to 
reenlist (ELGREUP1 cr ELGREUPS3). The final screer in 
setting up the AD data set included cnly those separated 
individuals who could be €asily identified by "ccecd" cr 
"bad" arterservice s€paration codes (ISC). Observaticas 
Keith seéparation codes in the "grey" area (death, hardskip 
Gischarge, entry inte officer programs, or medical disgquali- 
ficaticn) were removed from the data set since it was felta 
jJegitinate determinaticn of their success or failure could 
rot ke made. 

Eaving incorycerated these screens, freguency distri- 
kuticn analysis facilitated the removal of aberrant or 
impossitlée cases. For examfple, the maximum lengtk of 
service fretween 1 Sertember 1576 and 30 September 1982, the 
period of the data kase, was 72 months. Cases wke were 
jisted as having greater than 72 months LOS were rezoved 
from the data set. 


2. Cxreated Variatles 


This discussicn identifies the variables created in 
additicn tc those already in the master data base. Creating 
these variatles facilitated more detailed analysis of ckrser- 
Vaticne in the AD data set, and enhanced the enlistment 
standards mcdel develcpment precess. The following ccurents 
will alsc address tke reascns for selecting some variatles 
and nct cthers. 


a. Predictor Variakles 


There were several ways that individuals ir the 


Easter data base migkt appear in the AD rating during their 
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Gdie Gis They may have enlisted in a f~rogram to béccie an 
AD, taken tke AD rating exam, and/or achieved the AL rating 
through cn the job training. To distinguish between the 
Varicts ccmbinations of these frocesses, an entry group 
variakle (ENIRYGHEP) sas created. Freguency analysis cf tris 
new variakle ccnfirned that the final DMDC rate cf AD 
screened and selected oniy those cases who actually ended up 
as ADs. An effort was made to develop models for varicus 
combinaticns of these entry groups during stepwise recres- 
Sion analysis. However, the derived models were not signif- 
icant, and they did not improve upon the nodels ultizately 
selected for analysis. 

A common f[redictor variable in enlistment s¢tan- 
cards acdéels is cne dealing with education. The measure in 
the taster data kase reflecting education level (HYEC) was 
converted frcem a qualitative value to a continuous variatrle 
(CHYEC) to facilitate statistical analysis. In additicn, a 
dichctcacus (0,1) variable was created to reflect attairment 
ct a kigh school degree (HSDG). During stepwise analysis, 
which is discussed in the next chapter, each of tkese two 
New variatiles was offered separately as a candidate 
predictor variable. In nearly every instance, ESCDG was 
shown tc ke more sigrificant than CHYEC. 

Other couwmtcn predictor variables which measure 
entry-level attributes are ASVAB subtest scores. Ic allcw 


the use cf these measures of individual characteristics, the 


SCOLrES nere standardized, and the created variaklées 
(SASVABS) were considered during model development. As 
renticned in Charter II, standardized ASVAB subtest scores 


are used in varicus ccmbinations as composite measures. Cne 
cf these ccuposite variables 1s AFQT percentile (AFCTECNT), 
which alsc yields AFCI groups (AFYTGRPS). Another ccmrecsite 
is tke sccre used to determine eligibility fcr the AD 


Laeinc. Two variables were created in the AD data set to 
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identify this latter composite measure. Tic tipo taVaraarle 
created (ADCOMPOS) waS a continuouS variable which hada 
value egual to the sum of the four ASVAB standarcized 
subtest sccres that rake up the composite. Tne seccnd vaii- 
able created (ADMINSCR) was a dichotomous variable «hich 
distirguished those AICOMPOS values greater than or e€gual to 
190 frcem these ALCOMECS values less than 190. Each.time cne 
cf tkese fcur composite measures was offered as a candidate 
Fredictoxr variatle during regression model develcpurent, 
three separate trials were run. One trial containec the 
compcsite measure ard all i2 SASVAB variables. Anctker 
trial cortained the ccmposite variable and only these SASVAB 
variakles tkat did nct make uf the compcsite variable. The 
third trial contained only the composite measure with no 
SASVAE vériables. Additionally, the trials ccntained eitter 
AFQTFCANT cr AFQTGRPS, and either ADCOMPOS or ADHINSC:. The 
Furpcse cr this iterative process was to ensure multicclli- 
nearity effects were minimized among the independent vari- 
ables. During the cevelogpmert of the regression nodels, 
AFQTECNT and ALMINSCR were ccnsistently shown tc fe mcre 
Significant than AFCIGRPS and ADCOMPOS respectively. For 
this reascn, they xere included among the final candidate 
predictor variables tsed in sté€fpwise regression analysis. 

Another fredictor variable commonly ccrsicered 
ky enlistmert standards research deals with marital status 
and cépendents. Tke master file contains a qualitative 
variakle (METLDPND) tshich reflects marital status and numter 
of dependents. This study created a dichotomous variatle 
(DEPFENDIS) which distinguishes single individuals frem thcse 
who are gwarried and/cr who have children. Again an itera- 
tive frecess revealed this created variable to consistently 
ke mcre significant. 

The effects of race and sex were also considered 


in tke analysis by creating new variables. The best 


variatle inthe master file to indicate race and ethnic 
status identified categories cf whites, blacks and cthers 
(RACE). Since this variable was qualitative, three dummy 
variaklées were created (WHITE, BLACK, and OTHER). Tc allow 
analysis of the effects of sex, the master file variakie 
(SEX) was ccnverted tc a "0,1" variable (NUSEX). 

Several cther fredictor variables were ccnsid- 
ered and tested for significance and possible inclusicn in 
the final set cf cardidate fredictor variables fricr to 
develczying the regression models. Age at enlistment 
(ENTRYAGE), enlistgent paygrade (ENTRPAYG) and term of 
enlistmert (TERMENLT) were among those selected. Hany vari- 
ables were rejected kecause other measures were Fetter akle 
to capttre the desired effects. One particular variatle 
which did not show to be significant was the ccmpesite 
SCREEN variable (SCREEN) discussed in Chapter II. This may 
ke because the compcnents cf the SCREEN variable are indi- 
vidually mcre appropriate for analysis, particularly wken 
the e€mphasis 1S cn fredicting operational performance in the 
fleet. ‘Sinilar results were cited by UcGarvey [Ref. 10). 

The final set of predictor variatles created in 
the AL data set are interaction terms. These variables 
represent all twe-level interactions of the seven variables 
that met the specified significance level during stepwise 
regressicn anaiysis. The development cf these variatles is 


discussed in more detail in Chapter IV. 
k. Criterior Variakles 


As discussed in Charter II, this study usec two 
critericr variables when dévelcfing the six models--length 
ci sé€rvice measures and succéssS measures. The length of 
service measure for regressicn models is a ccntinuous vari- 


able (TAFMS1), and fcr discriminant models is a dichctcmcus 


KR 
NR) 


variaktle (SUCCTAF).2 SUCCTAF was assigned a value cf ‘one’ 
if the value or TAFYS1 was greater than or egual tc 48 
months, or if the value of TAFMS1 was greater tnan cr egual 
to 45 acnths and the individual entered the Navy in Cctccer, 
Kovemker or December 1978 Crh hho Na}. Tais was done to 
ensure those cases whe did not have the opportunity tc serve 
48 merths were nct iaprceperly classified as failures 
Individuals were considered as successes, for 


Eurpeses of this aralysis, if they served 48 mcrths cr 


longer, achieved faygrade E-4, and were recommended for 
reenlistment. Again, observations who didnot have the 
Sepportrniaty to serve 48 mcnths were also considered 


successful on the ICS porticn of this criterion if ttey 
served at least 45 mcrtas. The success criterion variatle 
(SUCCESSZ) captures these measures by considering SUCCIAF 
and twe cther created variakles (SUCCPAYG and SUCCREUEF). 

SUCCPAYG identifies those cases who achieved [E-4 
as measured by two created variables (PAYGRADE and NUHYFAY). 
FAYGFALE was created fron cne of two DMDC variables 
{PAYGKCDE1 cr PAYGRDE:) that measure an individual's faysgrade 
at the last file update or upfen discharge fren the service, 
aS affpropriate. NUEYPAY was created ky ccnverting ar NERC 
variakle (HYPAYGHRD) from a catégorical to a numeric vari- 
able. Usirg both DMIC and NHRC measures of paygrade ensured 
correct classificaticn of an individual on this ferticn of 
the critericn. 

SUCCKEUP, the eligibility to reealist pecrticna cf 
the success critericr, was derived from the DMDC variatle 
(ELGREUF1) that capttred the reenlistment code assigred ufon 
an individual's discharge frem the service. Service members 
ch active duty as cf the last master file update were 


considered eligitle tc reenlist, as long as there sas no 


2[iscrininant eae uire e 
x 


tegcrical 
vice continuous variatrles as” ass es. 


of ca 
variable 


cther iSicr taticr (ee chceweconeranye The next chapter wide 
discuss how the infcrmgation ccntained in the AD data sé€t xas 
specifically evaluated using three separate statistical 


analysis teéechnigues. 


IV. STATISTICAL ANALYSIS 

Three distinct statistical methods were employed in this 
research: Descriptive Analysis, Regression Analysis and 
LCiscrinirant Analysis. All methods used Statistical 
Analysis System (SAS) procedures to analyze the data and 
develcy the models. Table JI contains alist cf the 46 
candicéte fpredictorydiscrininating variables used in this 
study. In all, six sets of variables emerged, and each set 
was aralyzed using bcth regression and discriminant techni- 
gues fcr ccmparison. These€ six sets of predictcr/ 


discrininating variakles are shown in Table II, alcng with 


the apprcepriate criterion/classification variable. Each 
methcd, along with the results, are discussed in the 
fcllcwing sectiors cf this chapter. lies VOC imencting 


that, while the results may nct represent a marked improve- 
ment cver the selection process in use when individuals in 
the data set enlisted, the methodology presented may te 
applied tc further aralysis of the AD rating cr to any ctter 
Katirc in tke Navy. 


Ae CESCKIFTIVE ANALYSIS 


Descriptive analysis was accomplished through review of 
frequency distrituticns, summary statistics and multivariate 


correljiaticns. 
1. Freguency Analysis 


Freguency distributions are summary tables in which 
Gata are grouped cr arranged into conveniently estaklished 
numerically ordered classes or categories. The ¢reccess of 


data analysis is, tterefore, szade much more managearie and 


) 
in 


TABLE I 
Candidate PrecictoryDiscriminating Variables 


Label 


Variatle 
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frequency analysis wis perfcrmed 


In this study, 
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counts and percentage distributions of individ- 
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variables. 
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predictce 
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upon which tc screen 


Frovided a kLase 


Frequency distri- 


and tc ccmpare the results of this study. 


kuticns are provided in Appendix B for the AD rating. 
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statistics are 


tative data. 


like freguency 


TABLE Ii 
Selecticn Models 


distrirfutions, 
useful for analyzing and 


These summary statistics 


Predictcrs/ Reyression Disce iminant 
Discriminating Criterion Classificaticn 
Variakles Variable Variable 
TePENDIS TERMENLI TAFMS 1 SUCCTAF 
ADMINSCR H<SDG 
ELACK olor ty 2 
NUS EX 
TERMENLT INTERO3 Aros SUC GTAL 
INTERQG INTERO8 
INTER1G4 IJNTER21 
INTERO3 INTEROS8 TAFMS1 SUC CTAF 
SASVAEWK ENTHPAYG 
LEPENDTS #SDG PUCCE SoZ SUC GL oe 2 
CTHER TERMENLT 
SASVAEAL SASVABWK 
INTERO3Z INIEROY SUCCES o2 SUGGES = 2 
INTERO3 INIEROY SUCEESS2 SUGCEESS 2 
TNT Ea 21 Vela eRe 
SASVAEEI SASVABMK 
SASVAESI A2FCIGRPS 
CHYEC 
Variable sets A, E&, UT and E resulted fron 
stepwise regression technigues. The var- 
lable sets and F resulted from stepwise 
discringinart technigues. 
Table I prcevades the labels for these 
variables. : 
summary Statistics 


descriptive summary 
interpretircg guanti- 


represent frorerties 


cf£ ilccation, dispersion and shape, and may ke usec to 
extract and summarize features of the data s¢ét. 
Representative summary statistics for variakles in the AD 


Gata set are Shown ick Table III. 
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Selected Summary Statistics 
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Measuring the strength of the relationship Lretween 
variakles may be acccuaplished Ey correlation analysis. MTkis 
technigue e€rables one to gain an idea of the degree cf asso- 
ciaticn or covariaticn that a variable has with anctter 
variatle. The Summary measure that expresses the extent of 
this relationship is the coefficient of correlaticn, cy 
those values range frcm -1 for perfect negative correlation 
to +1 fcr ferfect. pesitiavewmeccrrelacnon. Values clcse to 
zero indicate little systematic covariation between two 
varialles. Correlaticn coefficients fcr guantitative vari- 
akles used in this study are ccntained in Appendix PB. 

Assessing the strength of association between vari- 
ables does not allow a resé€archer to predict the value of 
cne variaktle from the value cf another variable. The latter 
involves regression téchnigues, and is presented in the next 


secticn of this study. 
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E. REGRESSION ANALYSIS 


Regression analysis 1s cne method used t) develcp a 
statistical model that can predict the values of a dependent 
ck resztense variable based on the values of indepferdent or 
explanatcry variables. Rather than merely measuring the 
association between variablés with correlation analysis, a 
regressicn nodel attempts to predict or explain the value of 
the critericn variaktle by developing an eguation that is 
kased cn weighted values of one cr more predictor variatles. 

In déveloping the selecticn models in this study, the 
process emyloyed was to first apply a variable “search" 
procedure called stefwise regression. The resultant mcdels 
were ther aralyzed [Ey simple regression analysis, and vali- 
Gated acainst a hcid-cut sample of the data set. The 
details cf this process, the specific models derived, and 
results of the analysis are reported in the fcllcwing 
secticns. Appendix cC contains a discussicn of regression 


analysis assumptions and methcdology. 


1. Stepwise Regression 


Cne of the desired characteristics of a recression 
model is parsimony, hich means including the least numter 
cf explaratcry variakies that fermit adequate interpretation 
cf the dependent variable of interest. Such mcdels are 
€asier tc interpret ard are not as likely to be affected by 
wulticcllinearity? prcblems. In developing the aodels for 
this study, stepwise regressicn frocedures were employed to 
find a "best" combiration cf predictor variables, thereby 
avoiding the computationally complex and ccstly [recess of 


examining all possible regressions. 


3Multiccliinearity refers to the condition in which scre 
cf the inderendent variables are highly correlated with each 
cther. When multicoliinearity is present, the values cf the 
regressicn coefficierts for the correlated variables may 
fluctuate dramatically. 


Zo 


In this study, two sets of Candidate ~redicrcrm aes 
ables were analyzed nith the stepwise procedure. The first 
set included those entry-level attributes and measures tkat 
were cozrsicered likely tc Ere good predictors cf each 
critericn, rased on a review of similar enlistment standards 
studies. As discussed in Chapter II, these variables 
included individual and demcgraphic measures such as mental 
ability, amount of edcucaticn, entry age, entry paygtrade, 
Tarital status, AFQT fercentile, and ASVAB scores. tTakle IV 
Frovides a list of the 18 candidate variables from the AD 
qata set tkat were used in the stepwise procedure. 

The seccend set of candidate predictor variakles 
included the seven variables from the first set that met the 
specified significance level for inclusion in the stéfwise 
model. In addition, this set included all two-level inter- 
acticrns* of these seven variaktles. Inclusion of interaction 
terms in this study represents a marked departure fico 
previcus enlistment standards research. The results cf this 
analysis clearly irdicate the presence cf interaction 
effects a£ncng predictor variables. The séven fredictor 
variarkles and 21 irteraction terms used inthe stepwise 
analysis are alsc contained in Takle IV. 

Csing these two sets of candidate predictcr vari- 
ables, the stepwise procedure was run on each of the two 
critericn variables, IAFMS1 and SUCCESS2, waich were deéefired 
in Chapter VL; The resuiting four models were deéevelorged 
from a urifcrm randcmw ¢plit, the derivation sample, cf 1440 
cbservaticns in the AD data _ set. This derivaticn samfle 
constituted approximately half of the 2820 total cases in 


the AL data set. Sc doing facilitated cross-validaticumas 


— a a ee ee ee 


*An interaction ee ves ,the clo of twc cr mere 
independent variables, 1S included in regrelssicn modem 
when the relaticnshi pe oeeee one independent -variatle and 
the dependent variakle changes bon iffering values of 
another independent variable [Fkef. 11]. 
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TABLE IV 
Predictcrs in Stepwise Regressions 


Label 


Variatle 
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the validation sample, 


the mcdéels against a tolid-out sample, 


lnfluence the criciral 


not 


Characteristics would 


whose 


aa 


develcyment of the acdels. The predictor vaciatles ttat 
remaired in the model at the termination of the stefwise 
Frocedure were Significant at p < .10, and most variatles 
were Significant at 5~< .05. The four models thenselves 


Sere Slgniticant ates Une 


z- Multiple Regression 


The four models developed by the stepwise frecess 
were rext analyzed tsing the SAS Regression procedure to 
descrike the particular straight line model that test fit 
the data. Table V ccntains the printed output frem the SAS 
Regressicn jfrocedure that was run on each of the four 
rodels. Fcr comparative purposes, two models develcped by 
discrinuinant analysis technigues, discussed in the next 
secticn cf this chapter, are alsc shown in tarle V. MThe SAS 
User's Guide provides a detailed descripticn of the statis- 
tics that are included in the tables, as well as their 
nethcd cf ccmputation [Ref. 123. It can ke seen that Model 
E, with tte highest E-SQUARE and all variables statistically 
Significant, 1s the ;sreferred regression model. 

The propcrticn of variation in the critericn vari- 


able explained by the set of predictor variables selected is 


callea tte coefficiert of sgultifple determination, and is 
denoted F-SCUARE. The values of R~SQUARE for the mcdéels 
develcred in this study are relatively low. This may Le 


Factially attributarle to the large number of observaticns 
un the ALC cata set. However, it 1s also Jlikely that the 
variation cf the criterion variakle, length of Service or 
success as defined in this study, is also due to factors not 
captured Ly the entry-level attributes and measures used as 
predictor variables. These factors, which affect an indi- 
vidual's performance and decision to remain in the service, 
Fresent themselves subsequent to enlistment. They may 


include satisfaction srith initial assignment, gecgrafhical 
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TABLE V 
Regression Analysis Results 
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ccmmand climate, 


societal values and fres- 


change in marital status, 
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cpportunities cutside 


educaticral and economic 


and 


sures, 
the 


widitary- 


factors or measures are fest hoc 


These 


available when screening candi- 


consideraticns that are not 


enlistment and initial rating 


cates fox 


Trey 


assignment. 
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are issues that are appropriate for more sophisticated meth- 
cdolccies, for example, covariance structure analysis which 
can treat complicated enlistment standards models as a 
series of simultaneots eguaticns tnaat capture perfcramance as 
a "nultizle-stage" preccess occurring throughout an individu- 


al‘s military career. [Ref. 10] 
-3. WValidaticn 


The resuits cf the reéyression procedure were next 
valicatec acainst the hold-cut sample. Each of the recrecs 
Sion ucdels was derived frcem a uniform random sample, the 
derivaticr sample, cf the ckrservations in the AD data sé€t. 
The SAS Fegressicn j;rocedure was employed to calculate the 
Farameter estimates fcr the associated predictor variables 
using cata from cbservations in this derivation sample. the 
SAS S£ccre frocedure then used these estimates to predict the 
value crt the critericr variable for each cbservaticn ic the 
validaticr sample. Finally, these predicted values were 
correlated with the actual values of the criterior in the 
validaticn sample. MTIktese correlations represent the valida- 


tion ccefficients fcr each acdel, and are shown in Takle VI. 


Lap LEY © 


Regression Model Validities 


Mcdel PALS a 10 ie, Second Validity Average 
Coefficiert Coefricient Validity 

A Oar 4 e201 Oaza 

E Cals One oo O <2 

C O. 14455 0.13612 0.14 

t 0.1738% O.13' 766 0.16 

iF 0.17790 012759 0.14 

F 0. 14430 0213531 0.14 
Ncte: Tke First Validity Coefficient is the cesult cf 


tre cress-validaticn, and the Second Validity 
Coefficient resuits from the doukle cross-ss 
validation. The repecrted average is the single 
arithmetic mean. 


34 


PamL UG ENCE Gucecwemcr the validity 9£ the six 
regressicn models, the process was repeated by deriving 
Farameter estimates from the validation sample, and using 
these estinates to ccrrelate the actual and predicted values 
cf the critericn for cbservaticnos in the derivation samrfle. 
This dctkle cross-validation technique is descriked in 
detail Ly Campbeli [kef. 13]. fable VI alse contains this 
seconc set cf validity coefficients for the six modeis. 

Cccasionally, concern is expressed that trandon 
Samples may not be from a homogeneous populaticn, and, 
therefcre, the sample correlations may differ from the foru- 
Jaticn ccrrelations. One methcd of addressing the pfrecklen 
ef heterogereous samples is to average the correlaticn coef- 
ficients tc obtain a single estimate of the ferpfulation 
correlaticn. If the sample correlations are of akcut the 
same value and if tkey are not too large, asis_ the case 
with this study, this simple arithmetic mean will suffice. 
here this not the case, however, another téechniyue is te tse 
transfcrgations to Fisher's z coefficients. [ Ref. 145 The 
Simple arithmetic average correlations are also presented in 
Table VI. Appendix C contains the program used to calculate 


valicity coefficients. 


C. DISCBIBINANT ANALYSIS 


Ike third statistical method employed in this resé€arch 
was discriginant analysis. The use of discriminant analysis 
allows ckse€rvaticns tc be classified into two or more grotps 
cr categcries on the -asis of cne or more numeric variatles. 
As was done with regression analysis, the discritinant 
models were derived ard analyzed from the derivaticn sample 
of the data set, and tested against the hold-out Sanfle of 
cbservaticns. Variakles in the model were again selected 


uSing stepwise technicues. The resulting two models, and 


the fcur mcdels develcred by regression analysis, were then 
analyzed using the SAS Discriminant procedure. The pregran 
used in this analysis is contained in Appendix D, alcng with 
a discussicn of discriminant analysis assumpticns and 


VEC ueccrecy. 





1, Stelwise  Discriminane siaeyses 


The SAS Stepwise Discriminant frocedure was egfloyed 
to select tke most useful discriminating variables. It isa 
logical and efficient method of choosing an optimal ccmkina- 
tion cf variables. Their selection to enter or leave the 
model is Eased on either the significance level of an F teést 
cr a sguared partial correlaticn criterion. The selected 
variakles are those tshich ccntribute most to the discrinira- 
tory power cf the model. [kef. 12] 

The variables chosen Ey the stepwise discriminant 
Frocess sere selected from the 46 candidate variables shcwn 
Frevicusly in Tatle I. The entry-level attributes and mreas- 
ures that were consicered likely to be gocd fredictcrs, as 
discussed in Chapter II, represent 25 of these candidate 
variatles. The other 21 variables are the two-level inter- 
acticr terms considered during reyression analysis of the AD 
cata sé€t. The precedure was run on each of the two 
critericn variables, SUCCIAF and SUCCESS2, discussed: 
Chapter III. The criterion variables define the grours itrto 
whick each ecbservaticn will kEe classified Ly discriginant 


analysis, and are called classification variables. 


2. iLiascriminant Analysis 


AS previously mentioned, discriminant analysis 
involves the study cf differences between two cr ncre 
groups, defined by a single neminal level variable, witha 


set cf ccmacn discrininating variables. 
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fre SAS Discriminant Analysis procedure provideé the 
means for conductinc discriginant analysis cf the AD data 
set. The procedure sas run on each of the six models déevel- 
oped Ey stepwise regression and stepwise discrininant 
FLrocesses. Each ckservation is placed in the class fIon 
which it has the smallest generalized sguared distance. 
Also taken into acccunt were the prior probabilities of 
group membtershir. These prckabilities are obtained froma 
ireguency distributicr of actual successes and failures of 
the saazple cata set. This was considered ap;yropriate since 
this study 1s attempting to improve upon the selection 
Frocess in use at tke time the individuals enlisted. 

Tarle VII contains the results of discriminart anal- 
ysis. PAagmepreceanune Incocperated the prior prokakility cf 
group menkership, irdicated cn the classification matrix as 
PRLORS. The classificaticn matrix is divided into fcur 
elements which depict the nupber of actual (row) versus 
Fredicted (column) classifications into successful (1) cr 
failure (0) groufs. The fcur elements (actual, predicted) 
in the matrix are: 

(0,C) The number crt failure cases predicted tec te 
failures 
(1,C) The number of successful cases predicted tc 
ke failures 
(0,1) The number cf failure cases predicted tc te 
successful 
(1,1) The number of successful cases predicted tc 
ke successful 
Fach secticn first ccrtains the classification matrix devel- 
cped ty apriying the classification function to the deriva- 
tion samjle. The second classification matrix deéeficts the 
results of applying this same classificaticn functicr to 
cbservaticrs in the kcld-out sample, therekty validating the 
model. 


oo 


Ike table alsc shcws two rates relevant tc each 
classification matrix. The first rate is the percertace of 
correct classificaticnos, called the "hit rate", which 


Frovides 4 measure cf the accuracy of the discriminant 


model. The seccnd zate is the percentage of enlistees who 
were classified as (1,1) ccrpared to all cases whe were 
predicted as successful. It is called the "success rate", 


and it provides a measure of bow well this selecticn model 
would have rerformed. It may be compared to the cricinal 
selecticn strategy stccess rate, the priors. SUCCESS” Eau 
is an iupertant consideration with utility analysis) Wang 
will ke addressed further in Chapter VY. As with regression 
analysis, Model B is again the preferred model since it is 
the criy cne that improves ufon the selection strategy in 
existence during the timeframe of the AD data set. 

Jo illustrate how the results may be interpreted, an 
example cf the classification matrices for Model A will be 
explained. The model correctly classified 49 okservaticns 
as failures and 1079 cbservaticns as successful. The sun of 
these ccrrect classificaticns represents 73 percent cf the 
total cf 1440 observations in the derivation sample. To 
test the acdel's accuracy, the classificaticn functicn is 
applied to the validation sample. The secoad classification 
matrix irdicates 47 failure and 1039 successful observaticns 
were correctly classified. The sum represents a hit-rate of 
79 percent cf the total of 1380 observations in the hceold-cut 
sample. Tke ccnsistency of these hit-rates indicates the 
model is valid. Tke model Letters the 85 fercent success 
rate experienced by tke Navy with the selecticn process used 
at the time the cbservations enlisted. 

However, it 1s difficult to significantly improve 
upon such a high success rate. Additional entry-level 
attrikutes and sweastres might be found tc better carfture 


success as defined in this study. An alternate aprroach 


3§ 


would ke to redefine the success criterion. In either case, 
however, tte methodclogy fresented in this chapter tay be 
Similarly fcllowed tc develcr and test enlistment standards 
Bmodels. The next charter wili discuss a method by which the 
utility cf such an effort may Te measured. 
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Dascriginant Analysis Results 


Classification Matrix Hit Suceces 
Rate Rate 
EFredicted 0.78 Q2ee 
SUCCTAF 
C 1 Total 
Actual 0 4g 16 1 210 
SUCCIAF 
te 1079 1230 
Tota 200 1240 1440 
Predicted QO.79 QSteee 
SUCCTIAE 
Q 1 Total 
Actual 0 47 136 183 
SUCCTAF 
158°) 1038 1197 
Total ZED “Wiis 1380 
Predicted 0.85 Q285 
SUtCCTAF 
Q 1 Total 
Actual] Q 1 209 210 
-<UCCIAE 
1 Zz 1228 1230 
Total a 1437 1440 
Predicted 0.87 Otay 
SUGCTAFE 
Q 1 Tota 
Actual 0 0 183 183 
SU Ce iar 
1 1 1196 11$7 
Total 71 1379 1380 
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Vo Ulery ANALYSTS 

This chapter cortains an explanation of tae apfrlica- 
bility cf utility analysis to the development of selection 
procedures, and discusses the theory of utility analysis. 
The metkcdclogy used in this study to apply utility analysis 
is described, alcng with secticns on the calculaticn cf cell 
prokatiities for regression and discriminant models, anda 
secticn cn estimatince cell utilities. More detail cn the 
calculaticns and procrams used for utility analysis may be 


found in Appendix E. 


Aw FUBECSE OF OUTZLITY ANALYSIS 


The development cf a mcdel for use in predictirg an 
applicant's future performance in a particular jok is a very 
necessary fart cf most selecticn procedures. However, the 
model itself does net constitute enough informaticr to 
enatle a decision to be made on whether or not it is worth 
implementing. The validity of the model is one indicatcr cf 
its fetertial usefulness but, as will be seen, other factcrs 
Significantly affect the usefulness of a model. All crcani- 
Zaticrs sould find it valuakle tc be able to judge the worth 
cf their strategy in guantitative terms, particularly wien 
comparing their existing strategy to a newly develcped, 
competing strategy. A framework is needed which will allcw 
the evaluation of a selecticn model in terms of the institu- 
tional gains (or losses) that are expected to result when 
that aucdel iS used to guide decisions on  selecticn. 
Classical utility analysis provides such a framework, ard it 
allews tke calculaticr of usefulness to be made in terus of 
actual] dcllars, whico facilitates the comparison cf cne 
selecticr mrcdel with another. 


<< 
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En. IEECKY CF UTILITY ANALYSIS 


In the context cf utility analysis, there are fcur 
cutccues cf interest associated with selection decisiogcs. 
ihese cutccres are; 

« Valid Positives (VP), which refers to the numrer 
cf applicants that are hired and whe turn cucece 
re successful cn the jot. 
« False Positives (FP), which refers to the numrer 
cf applicants that are hired and who turn cut to 
ke unsuccessful on the jek. 
« False Negatives (FN) are the people who were net 
hired, but wke would have been successful if 
tkey had teen Lired. 
¢ Valic Negatives (YN) are the people that were 
net hired, and who would have peen unsuccessful 
if ttey had Leen hired. 
It is ckvicus frem tke termincicgy and the explanaticns that 
VP and VN cecnstitute correct selection decisions, and FE and 
FN represent selecticr error. 

Trese cutcomes are perhaps €asier to understand with the 
aid of a diagram. fFicure 5.1 shcws the relationshir Lreétween 
hypothetical predicted (from a model) and actual scores ona 
job rerfcrmance crité€rion ificr a large number cf job 
applicants. 

The eljJipse contains the data on predicted and actual 
GCLRECTICn SCOkes. In this diagrammatic example, the ccrre- 
daticn Ltetween the predicted and actual scores (the mcdel's 
validity) 41s apfarent--higher predicted scores are associ- 
ated aaitk higher actual scores and vice versa. The Ecint 
cn the vertical axis is the dividing line between what is 
consicered to be successful performance (say completicn cf 
48 months cf service for first term enlistees), and ursuc- 


cessful ferformance (less than 48 months service téefcre 
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Figure 5.1 Hypothetical Predicted and Actual Sccres 


discharge). In utility analysis the term base 


ir 


ate is 
defined as that pregertion of current employees whe are 
consicered to be successrul. If seven out of every ten 
emplcyees are successful, then the base rate is .70. The 
meent X Cn the horizcntal axis 1s referred to as the cut 
Score. If an applicant's predicted score (from the mrcdél) 
1S gxeater than x, then that person will be accefted 
(hired), ard if theiz predicted score is less than x, tten 
they will re rejected (not hired). The iocation cf x cr tae 
horizcntal axis will cften defend on the selection ratio, 
which is the propcrticn cf applicants that need tc te 
accepted in order tc fill a certain number of jeks. rc, 


cver tke ccurse (ef che years 80 job vacancies are expected 
to occur and if 100 agplicarts over the year are expected to 
apply fcr those jobs, then the selection ratio needs to be 
-80 if all vacancies are to te filled. In the happy CGinreuge 
stance (from the recrriter's pcoint of view) where there are 
far ucre aprlicants tkan joks, then the cut Score wa tee 
dure. Gtility is defined here to mean the expected gain in 
dollars that results from a farticular selection strategy. 
Tre lines generated from the Ease rate and the cut sccre 
divide tke sample irto four cells as Shown. Each cell 
contains the peorle who are classified into each of the fcur 


cutccnaes cf interest. In celis 1 and 2 are people x«hcse 


predicted score is higher than the cut score. Therefcre 
these fecple would be classified as accept. These accepted 
Feople (tke fositives) are further divided into these sho 


would ke successful (valid fositives) and those who wculd be 
unsuccessful (false [csitives). Cells 3 and 4 contain the 
Feople wke scored lower than the cut sccre cn tne predictcr, 
and tkese wculd be classified as reject. Again, sce of 
these rejected cases would have been successful (false 
negatives), and soze would have failed (valid negatives). 
In utijity analysis it is ccnvéenient to convert the celi 
counts (represented [Ey VP, FE, FN and VN) tO propcrticheoen 
the cverall sample, <¢o each ccunt is divided by the rumter 
cf pecple in the sample and the cell prebabilities (PVP, 
FFP, EFN and PVN respectively) result. 


Cneé further result of interest is the su 


ccess rate. the 

success rate is defired as the proportion of Ritedeapea 

cants whc are, or wiil be, successful. It is simply fcurd by 
dividing FVE by the stm of FVP and PFP. 

Giver the concerts and terfinology outlined above, it is 

now fessikle to disctss in general terms the factcrs that 

will affect the cell rrobabilities which, in turn, Sareea. 


the expected utility. 
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i. Yodel Validity 


The model's validity, as measured Ly the correlation 
ketween predicted and actual scores, 1s one factcr tkat 
detergines the degree of selecticn error resulting fren the 
selecticr strategy. Pietne Validity is high, then the 
Broperticns of correctly classified people (PVP and FYVN) 
will ke tigker, and the selection error (PFP and PFN) will 
ke lewer. Vineberg and Joyner in their review of almcst 150 
Wilitary studies related to job performance fredicticn, 
found that validities range frem .15 to .40, from a tctal of 
350 validity coefficierts [ Ref. 15]. Generally, valicities 


withir this range would be considered as low or mediut. 


-~ 


<- Ease Rate 


If the existing base rate is high (say .70 or 
greater), then it means that whatever selection stratecy is 
currently in use has a high rate of success in identifying 
Fotentially successful apflicants. Under these circun- 
stances, it is unlikely that using anew model in the 
staffing process would yield much of an improverent in 
correctly selecting ajgrlicarts. A high base rate means that 
the cell] frobabilities for FVP and EFN are going tc be 
bighexr tkar for FFP ard PVN. 


dad 


- Selection Ratio 


Assuming the acdel is valid, the lower the selection 
Tatic, tke more useful the rodel wiil be in identifying 
successful applicants. Decreasing sélection ratics mean 
that the organizatior can be increasingly selective in whom 
it hires. Naturally, it will tend to accept only these who 
score highest on the fredictor, those who are also predicted 
most likely to be successful. A low selection ratic (high 
cut score) will mear that PVE and PFP will be small. bt 


also fcliows that a lcw selection fratic will yield a Higres 
success rate--althouck few pecple will ove hired, mest of 


them will represent ccrrect selection decisions (PVP). 


Cw. ESVJIBATING THE UTILITY CF A SCDEL 


The expected utility (EU) cf a model is found by suugming 
the products of each cell prekability and its asscciated 
cell utility (Ui, 02, U3 and 04), and Subtracting theveoa, 
c£ giving tke test t¢ an applicane ie 


EU = C1I(EVP) + U2(EFP) + C3I(EFN) + U4(PVN) - UT (5 «le 


Apperdix E contains detailed descriptiors on how cell 
Frokarilities and cell utilities are determined. FE Cl ee 
discrigirant model tke cell pickrabilities may be readily 
derived frcerm the output of the SAS Discriminant frecedure, 
kecause tke model classifies cases into predicted successes 
and predicted failures. In the regression model the cut 
score is net kncwn ir advance, so cell frokabilities that 
result frcenr a number cf possikle cut scores are calculated, 
and a cut score is eventually chosen based on which set of 
cell yickabilities maximizes the utility of the model. 

The fornula for calculating the expected utility ofa 
model recuires that a utility be assigned to each selecticn 
cutccte. These cell utilities are designated U1 thrcugh UY 
and are associated sith the cutcones VP, FP, FN and VN 
respectively. The Billet Cost Model provides an estinate of 
the ccst tc the Navy cf staffiny a billet. In this study it 
is assumed that this ccst is egual to the marginal preduct 
cf a successful sailcr, and so the utility of a valicg ocd. 
tive (UT1) is assigned a value of $24,163 [Ref. 16]. No 
Froven technigue exists for estimating the cell utilities 


for the three other selection outcomes. Individual 


We 


circuaustances and jyrevailing market conditions take it 
difficult tc estimate these cutcomes with real confidence, 
eo these cell utilities were estimated relative to U1, anda 
mincr fcrm cf sensitivity analysis was conducted. The cell 


utility cf a false pesitive (Uz) was assigned values cf -.5, 


alreagd — 2. Valid necatives (04) were assigned an egual and 
cppeosite utility to U2, and false negatives (U3) were 
assigned values cf 0, -.25 and -.5. Table VIII shows seven 


different sets of cell utilities that were considered. 


TAELE Vili 
Relative Cell Utilities 


U1 U2 U3 U4 
de. 0 Ue) Ore) Olas 
1. 0 a Oran 0.0 Oe 
‘benO -1.0 O20 1270 
te) i one O10 Zen 
1. 0 -0.5 — Or) Qa 
15:0 tie. 0 aa a) 0 
a. eed 0 o> Zane) 


Tre ccst of administering a test (UT) is of significance 
if the ccsts of testing are different for ccmpeting selec- 
tion strategies. The models developed in this study tse 
cata cattered from the existing tests, and therefcre the 
costs cf testing will remain ruch the sane. Thus in this 
context, UI may be icnored since it applies ejyually te the 


cld ard rew tests. 


De RESULTS OF UTILITY ANALYSIS 


Tarles IX and X ccrtain the results of the utility anal- 
ysis cr the regressicr and discriminant models respectively. 
The “Percerct Change in EU“ cclumn is the result cf the 
compariscn cf the mcecel's utility with the utility cf the 


Navy"s criginal selection strategy (base line utility A 
Fositive percentage change in EU indicates that the Lraxinun 
utility ckrtainable trem the acdel is higher than the utility 
cf the criginal selection strategy. An increase in utility 
cf say $£0 means that the Navy saves $50 for each selecticn 
decisicr (ccrrect or incorrect) that is made by using the 
model rather than tte original strategy. For the rcdels 
with the SUCCTAF or TAFMS1 criterion, the base rate is .8€1, 
14.e., €€.1 rercent cf the fecrle selected by the Navy were 
successful. These jyecple can be thought of as tke vajid 
Fositives ci the original strategy and the remaining 12.9 
Fercent are false fecsitives. (For the SUCCESS2 Critepaem 
these figures are 76.8 percent an GUemeceer percent.) 
Unfortunately it is nct possiktle to calculate the values of 
false ard valid necgatives so these are considered tc be 
zero. For the TAFMS1 or SUCCTAF criterion then, the jeu 
Froktakilities for the original selection strategy are FVE = 
-861, PFE = .139, PFN = 0 and FVN =0. The base line utility 
for each cf the three differert combinaticns of U1 and J2 
can then re calculated. The model utilities are tten 
compared tec these base line utilities and the differences, 
expressed as a percentage of the base line utilities, are 
reported. Similarly the base success rate cf the criginal 
strategy is also .861 (for the TAFMS1] or SUCCTAP critcraonme 
The cclumn "Change ir Succrate" rceports the actual diftter. 
ence retsheen the models' success rates and the base success 
rates. The column "SRATIO" shows the selection ratic that 
results wker the cut score is chosen so as to maximize the 


utility, iczr each set of cell Wtplitiecs. 


ales hRearessicn Mcdels 


=a EE ae See ee eee —m™e e-em eo 


Fer most séets of cell utilities, the regression 
models developed shcw little improvement over the criginal 


selecticr strategy. In mcst cases the selection ratio is 


50 


very clcse to land the percentage increase in expected 
utility is very small. Thi¢ is not a surprising result 
kecause the model validities are relatively low (arcund .2z0) 
and, more significartly, the -Lase rates are very hich at 
-861 and ./68. It is interesting to note, however, that 
when tke ccsts cf a false positive and the benefits ofa 
walid necative are hich, then the selection ratio is driven 


down, and tke utility and success rate go up. 


In general the discriminant models did net yferfcrn 
as well as the regression medels or the Navy's cricinal 
selecticr stratecy. For scme mcedels the percent change in 
EU was a Significant [fositive numter, but these were usually 
associated with extreme assumpticns of cell utilities. In 
additicr tc the factcrs menticned in the previous sutsec- 
tion, this peor performance is because the discriginant 
models lack the flexibility tc vary the cell protatilities 
dependinc cn the valtes of the cell utilities. There is no 
cpticn tc vary predictions depending on the conseguences of 
correct and incorrect selection decisions, and thus cniy cne 
set cf cell prokabilities is available for each discriminant 
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VI. CONCIUSICNS AND RECOMMENDATIONS 

This study set cut to prcevide a method foc develeping 
enlistmert standards models which improves upon sigilar 
processes fFresently ir use. Toward that end, significant 
advances have been made, particularly when compared te fricr 
studies conducted at the Naval Postgraduate Schocl. ihe 
technigues tsed provide a much more comprehensive afrroach 
to medel develorpment. They emerloy regressicn analysis to 
fully develop the stepwise regression results. In additicn, 
stepwise discriginant procedures were used to find an 
cptinal mcdel pricr to full discritimare analysis. 
Alternative criteria for measuring successful operational 
perfcri@arce, ancluding a ccntinuous length of service 
critericr, were incorrorated in the models. Finally, “eae 
model was analyzed tsing Ecth regression and discriminant 
analysis technigues. 

Ferhaps most sigrificant is the presentation of a feans 
Ty which the benefits from such efforts may be gauged. The 
develcrreat of innovative utility analysis prograns affords 
future researchers ar excellent opportunity to measure in 
monetary terms the benefits tc te derived from implementing 
anew selection strategy. it 1S important to reiterate that 
the statistical and ttility analysis technigues presented in 
this study may be easily afrlied or modified to acconmncdate 
selecticr standards model development for any of the mcre 
than £Q Navy ratings ccntained in the master data base. 

A seéeccnodary purgese cf this study was tc ciscover 


whether the models develcped ifprove upon existing selection 


and assignment stratecy for the AD rating. By and larce, 
the wmodeis presented do net appreciably enhance the 
Processes used since 1976. The models do, however, allow 


cne tc fccus on some specific considerations in the current 
screening processes. For exarple, Models A, B, and C allow 
rolicy takers to consider length of service in months, and 
to vary the critericn for measuring success. This cafa- 
Privty a2 farticularly appropriate for use in a dyfraric 


recruitizg wtarket. 


Ae. EESUITIS 


This study yielded several cther results worth necting. 
The tern cf enlistmzent variable may be used tc fredict 
success now that it has been corrected to reflect active 
fey «6 CE Tic ation. This is particularly important wken 
assessing Naval Reservists, whose six year contract cener- 
ally xreguires only three years of active service. The 
change frcnz a negative to a significantly positive ccrréla- 
Plonect IERMENLT on the criteria is one of the mcre iagpcr- 
tant discoveries of this research effort. 

This study also determined that the usefulness of the 
SCREEN ccuapesite sccre in predicting job fperformance reéas- 
uceS was virtually ronexistent. It appears tc ke mcre 
apprcgpriate to use the SCREEN score components ir the 


todels, at least when attempting to predict cperaticnal job 


performance. Nentraditional ASVAB subtest scores, such as 
Auto Infcrmation, hay alsc be appropriate for use in the 
screering process. A@@reCramsolgnificant finding ci tkis 


study is the definite presence of interaction effects. 
Considering perscnal measures on an individual ir ccnjurc- 
tion with cther measures represents a marked change in 
current selection practices. 


Ic suMmarize the results of the statistical analysis, 


the variafles measuring term of enlistment, educaticn, 
dependents status, sex and race emerged as refeéeatecly 
significant predictors Cr successful Operational 


ECEFCIERALCE. The ccuposite measure of eligibility for the 
AD rating, and the ASVAB Auto Information subtest eccre, 
were alsc significant fredictcr variables. In additizeme 
Model EF was shown to ke the best regression and discrigzinant 
todel. 

The results cf the application of utility analysis stow 
that the regression acdels developed in this study ~ertenm: 
as well as or better than the original Navy stratecy which 
was used as the coazarison (rkase line utility). ita 
important tc note however, that the methodology used in this 
Fart of tte study ensured that regression models will 
Provide a maximum utility at least egquai to tne Ease line 
utility. This is because the techniyue allows tae cut sccre 
to ke set sc low that all cases are accepted. Models A and 
F are corsicered tc kre the Fest of the models because ttey 
Frovide for significart increases in utility without having 
to resert to ampractically low selecticneeralee-— The 
discrigirant Models A and E are better than the ctnecs 
kecauseé improvement cver base utility is possible, depending 
cn the cell utilities. 

As was mentioned in Chapter V, the high existing base 
rates are an irdication that newly develored medels are 
unlikely tc prodtce superior results. Utility analysis is 
hindered Ey the difficulty cf confidently estimating the 
individual cell utilities, and this is an area that is in 
need cf further research. It ais also difficult tc ccenupage 
new selection stratecies to existing ones because it is 
impossitle to classify the cases rejected by the existing 
strategy as valid or false negatives. Data of this scrt can 
cnly kre citained [Fy testing all applicants and then 
accerting all of them, regardless of their relatiorSship ge 


the cut sccre, or to the desired selection ratio. 
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Es KECCKMENDATICNS 


Lespite the advances made ty this study, there remains 
Many cppertunities tc refine the models presented for the AD 
BaAti Lg, and to develop models for cther Navy ratinc¢s. 
Frocecurally, these cfrortunities include testing fcr curvi- 
linearity cf the models, expanding the interaction terms to 
three cr mere levels, and seeking different combinaticns of 
ASVAE suktest scores aS pctential predictors. There may 
also te cther measures not evaluated by this study that are 
significant operaticnal performance predictors, suck as 
enlistmert saivers, [EP status, or involvement with caival 
Biltnetities. 

Ccnsideration skculd alsc be given to altering the 
criterion variables. One particularly promising ac<ustment 
May ke te change tke critericn to reflect achieving E-5. 
This way be appropriate since the models developed aprear to 
do a fetter job of fredicting lcnger LOS, as indicated ry 
prelininary residual analysis. Developing separate .rcdels 
that yield predicticrs of shorter LOS may also be in crder. 

Tke multiple-stage analytic approach referred tc ina 
Chapter 1V¥V also éppears to tbe a promising technigue. Suei 
analysis gight ccnsider change in dependent status, ;jerforn- 
ance evaluations, or advancement exam results as variakles 
in a ucdel. 

Ic improve the tesefulness of utility analysis it is 
impcrtant that ateckniyue Le developed to estimate cell 
utilities with reascrable accuracy. Such a technique needs 
to ke akle to control for charges in the recruiting nmarkét, 
and fe sé€nsitive tc the changing Navy reguirements for 
recruits. It is also impertant that data be gathered on 
applicants who are nct accerted into a particular ratins, tec 
aliow researchers to determine if they were reclassified to 


another rating, or rejected entirely. 
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ip ccnelusion; it is clear that continued eGLicricwa. 
develcr selecticn standards mcdels ror ail ratings are 
essential. For it is througn these efforts that the vecerme. 
training ard maintaining Navy personnel wiil be reduced. 
The resultant experienced career torce will ensure the Navy 


is ready tc meet any glotal ccumitaent. 


ae 


DATA EASE DEVELCPMENT PROGRAMS 


This appendix prevides the SAS programs used in this 
study to access the master data base, develop the AL data 
set, anc create new predictor and criterion variatles, as 
Giscussed in Chapter Iil. Each program contains the job 
ccntrel language information appropriate to tne Naval 
Fostgraduate Schcol's IBM 3035 computer system. Statistical 
Analysis ‘System (SAS) statements are employed ir the 
Frograms tc acccmplish the desired functions. These SAS 
statements are normally preceded by comments tc eExflain 
their furpese, the ccaoments Feing identified Ly an asterisk. 

Takle XI cortaixrs the fregram called “ADSETUP". This 
Frogram was used to access the master file and extract 
infornaticr on Aviaticn Machinist's Mates (AD). (Ine gaster 
file tare, Originajily called "ENLIST", has receatly been 
revisec and relakteled "NPS709".) The data file created by 
this pregran is called "ADDATA™", and it contains the iritial 
z43 variatles frem the master file. Also frovided in the 
Erogran are the variakle names and labels. The prograa aay 
Fe used to extract data from the master rile fcr any of 
apprcximately 90 Navy ratings simply by entering the ajfyro- 
Friate akbreviation and four digit code for the selected 
matinc. 

Table XII provides the program called "“ADSCREEN" tkat 
was used tc screen tte data extracted from the mastexr file. 
These screens were performed cn cbservations in the "APLATA" 
trie , and the results were placed in a file called 
PaO SuESEI". Because of the large number of cases and vari- 
ables in the data, sufficient computing work space was not 
availacle. Therefore, the SAS KEEP statement was used to 


retain 116 ci the iritial yariables f£OrEweana ss = It was 
felt tkese 116 variaktles captured all the desired treasures 
co the ckservations that would be reguired for analysis. 
The last screen was ifcorporated following frequency distri- 
kuticn aralysis to renove cases that had aberrant cr inpes- 
sible data associated with then. 

Takle XIII ccntairs the fregram called “ADNEWVAR". This 
Frograg was employed to create new predictor and criterion 
Varliatlese, aS discussed anv Cha tenn am The pregtrag used 
jnformaticn on observations in the “ADSUBSET" data fi ]euee 
create tke new variatles, and placed the results cf these 
cperaticrs ina fide called "ADALL4". This file tkus 
constitutes the AD data set referred to throughout tris 
study. It contains all of the selected and created vari- 
ables that provide irformaticn on the 2820 ADs who reraired 
in the cata set after all screens were accomplished. lt as 
this file that was used to ccnduct the statistical analysis 
for, thie .ctudy. 

The “ACNEWVAR" pregram lists all created variable names 
and 9 iabels. It also contains the SAS statements that 
converted several gualitative variables to numeric variatles 
ck dichctcnrcus (0,1) variables. Finally, the prograr shcws 
the SAS statement usec to split the AD data set into the two 
unifcrmly distributed randcm samples (RANDALL1). These 
derivaticn and validation sagrpfles were used during regres- 
sion and discriminant model development described in Chapter 
aS 
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CESCRIPTIVE ANALYSIS RESULTS 


Freguency distrifutions and correlations used for 
descriptive analysis cf the AD data set are csortained in 
Tables XIV and XV. 

Ike frequencies stow that $2 percent of the AD cata set 
were 17 tc 21 years cf age, 79 percent had a higt schcol 
degree, $7 percent were single, and 98 percent were male. 
Even thcugh BLACK ard OTHER oniy fcrepresented 17 and 6 
percent cf the sample respectively, their criterion scores 
were Significantly different compared to WHITE criterion 
scores. Thus, BLACK and OTHER emerged as predictors in scme 
cf the mcdéls. It is interesting to note that 40 fercert of 
the sampdie achieved tle paygrade E-5. Using achic¢vement of 
EF-5 rather than E-4 ir the ccmposite success critericn would 
Froduce greater variability on the criterion which may 
imprcve tke models. 

Cne third of the cases in the data did not score 190 or 
greater cn the AL ccomposite sccre. These cases are e€itker 
Feople wko were classified pricr to correcting the ASVAB 
Forms £,€ and 7 misncrtming proktlems, or people whe rigrated 
to the AD rating stIseguert to service entry. This gay 
Fartialiy explain the negative correlations these variables 


have witk tke criteria. 
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TABLE XV 
Selected Correlations 
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PN Tea 


INTER ZO 


INTER2Z1 


Ncete; 


0.00629 Seg 271 Ce Ge s7 
0.7384 Ghz 250 0.3845 
~0.05623 =O.02818 ms Ue Ved 
O200ZE 0.1347 Giz 4 £2 
-0.08291 —O.Us7e0 -0.028€3 
0.0001 0.0443 Oe 
The first number is the correlation be: ween 


the predictcr and the criterion, the seccnd 
number is tke significance level. 
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BEGRESSION ANALYSIS PROGRAMS 


Recression alalysis attempts to predict or explain the 
values cf the critericn variable with one oF mote ~redicram 
variatrles. The follcwing sections expand upfon the discts- 


Sion cf recressicn aralysis presented in Chapter IV. 


Aw. FBECUJREMENTS AND ASSUMPTICNS 


Wwker ccrducting regressicn analysis, certain requite.s 
ments grst Le met or assumed. One of these requirements is 
the use of guantitative variatles.5 Application of recres- 
sion f;recedures alse reguires normality (the value of the 
Gependent variable must be normally distributed at each 
value of tke independent variable), homoscedasticity (the 
variaticr around the regression line must bre censtant foc 
all values cf the incependent variable), and independence of 
error (tke residual difference between an observed and 
Fredicted value cf tke dependent variable must ce indefen- 
dent fcr e¢ach value cf the predictor variatle). Anctserc 
Ileguirement of linear regression is that a straight-lire cr 
linear relationskip €xist between each independent variatkle 
and tke dependent variable. For purposes of this study, and 
Fased cr initial irvestigaticn, these reguirements are 
assumed to Le met. Ecwever, an e€xtensive effort tc €valuate 
these assumptions by transforming the variables or emplcying 
ccmplex statistical analysis packages has onct been 


conducted. 


STbe inclusicn. of gualitative or oye eee variatles 
in recressicn models may be accommodated through the use of 
dummy variarles. 
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Epes Leber SE REGRESSICN 


The SAS Stepwise process considers each of the candidate 
independent variables for inclusion in the model by deter- 
Bininc tke contribution the variable makes to the _ nicdel. 
This deternination is acccmuplished by calculating the 
Fartial F statistic fcr the variable, and adding it tc the 
model if it meets tke specified entry significance level. 
After a variable is added, the stepwise method then iccks at 
all the variables in the model and deletes any variakle that 
does net previde an F statistic sufficient to meet the spec- 
ified sicnificance level for remaining in the model. This 
Frocess of adding ard deleting variables continues until 
none cit the variables has anF statistic significant to 
enter cr leave the mccel.©® [Ref. 12} 


Cs. LINEAR KEGRESSIO“R 


Simple linear recressicn is concerned with findirg the 
statistical model or eguaticn that best "fits" the criginal 
data. This is accomylished Ly defining a straight line tkat 
Hininizes tre differences Letween the actual value of the 
dependent variable and the value that would be predicted 
from tke fitted line of regression. The SAS Regressicna 
Frocedtre uses a matkematical teéechnigue, the least-squares 
mwethced, tc produce such an eguation for the best linear 
todel. This eqguaticn provides the intercept and slcfe of 
the samuzple predictor variable. With multiple linear regres- 
sion, tkese€ slopes represent the unit change in the derpen- 
dent variarkle per urit change in the independent variakle, 
takirg into acccunt the effects of the other indeperdent 
variatles, and are referred to as net regression cceffi- 


cients. The sample regressicn ccefficients of the rredictor 


his eae usec the SAS Stepwise default significarce 
e-15 £0r variakles to enter or remain in the model. 
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variakles are then used as estimates of the respective foru- 
laticr farazeters. for al lustratrony the program used to 


validate Model A is frovided in Taple XVI. 
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APEENDIX D 
DISCRIMINANT ANALYSIS PROGRAMS 


Liscriginant Analysis allcws observations to be classi- 
fied intc two or more grours cn the basis of one cr mere 
Numeric variables. The following sections expand ufen the 
discussicn cf discriginant analysis presented in Charter IV. 
For dailitstration, Table XVII shows the program usec to 
Froduce tke classification matrices for the derivaticn and 


valicaticr samples fcr Model A. 


Ae FECUIREMENTS AND ASSUMPTICNS 


As was the case with regression analysis, 1iscrizinant 
analysis also reguires that certain basic assumptions be 
met. First, the ckservaticns in the data set skculd be 
members cf twe cr more mutually exclusive grcurs., 
Therefore, the groups must te defined so that each case will 
Kelon¢c to cnly one group. Another statistical [frofperzty 
reguired of discriminating variatles is that they may nct be 
linear ccmtinaticns cf other variables. Thus, the €un oc 
averace of several variables may not be used alecng with 
those variatiles. Ikere are three other assumptions to be 
considered. The pepulaticn covariance matrices must be 
egual fcr €ach group, each grcup is to be drawn frou ae for 
laticn which has a sultivariate normal distributicn, and 
discrigirating variatles must re measured at the interval or 
Tatic levels. Ideally, these variables will be ccntinuous, 
Fut tkey need nct Le. [Ref. 17] This study assumes these 
reguirements have beer met. However, an effort to €valuate 
these ficperties was ret conducted since, in practice, the 
discriginant analysis technigue is rather robust and can 


tolerate scne deviaticn frog these assumptions [Ref. 18°. 
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Es. DIsSCRIBINaENT ANATic@s 


Tke first step cf discriminant analysis is to weigit and 
linearly ccrkine the discrigirating variabies so that the 
grours will be as Statistically distinet as possi Gee The 
derived equations, called discriminant functions, ccmtbine 
the c¢rcetp characteristics using a measure of generalized 
sguared distance’? that will allcw one to identify the cercup 
to which a case Felcncs or nost closely reseatles. 

The classificaticn process may assume that memberstir in 
@groupr tas egual likelihood cf occurring. However, it may 
ke mcre desirable tc incorperate the prior probability of 
group membership intc the classification function tec impreve 
Fredicticn accuracy or minigzize the cost of fprediction 
errors. In this study, membtership in a success gLrcuUEZ was on 
the crder cf 80 fercert. Therefore, it was approfriate to 
consider prior frobakilities so that those cases predicted 
as unsuccessful would be classified as such only if streng 
€viderce exists that they belcng there. 

The ultimate ccncern in developing a mathematical rodel 
is that it predict well or previde a reasonable déscriftion 
cf the real world. Cree a acdel is developed which frevides 
satisfactery discrimination for cases of grcup memkership, 
classification functicos may tre derived and applied tc the 
classification cf neéw cases with unknown group membership. 
A gocd test of the adequacy and accuracy of the discrigzinant 
model is the percentace of correct classifications, ccimorly 
called tke “hit-rate™. This test 1s accomplished by 
applying the classification function to the known cases fren 


which tte functicn was derived. The percentage of ccrrectly 


7JThe frocedure ccnducted a likelihocd ratic test of 
homcgeneity of the within-grcur covariance matrices for each 
todel. This _test was statistically Significant fcr each 
model. Therefore, the within-group matrices were uséd as 
the Easis cf the measure of generalized squared distance in 
develcring the classificaticn criterion. ReLs eel 


$0 


classified cases provides an indication of the accuracy of 
the procedure and irdirectly ccnfirms the degree cf grcup 
separaticn. The results may ke depicted in a classification 
Ratrix. 

kKhen the sample size is large enough, as it is ir tkis 
study, a further check of the classification accuracy may be 
conducted Ey randomly splitting the sample into two sukseéts. 
“The classification function is derived on one subset and 
valicated on the otter sutset. A comparison of the two 
hit-rates f~rovides the measure of accuracy of the acdéel. 
[Ref. 17¢ 
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APEENDIX E 
UTILITY ANALYSIS PROGRAMS 


This appendix prcvides further details of the irficria- 
tion ccrtained in Charter V, and gives examples of the SAS 


Frograms and outputs. 


Aw. CAICULATION OF CEILI PROEFAFAILITIES 


The method used tc calculate cell probabilities in tkis 
study depends on wtether a regression ora discriminant 
model is being evaluated. A regression model can Le viewed 
Simply as a fcermula for calculating predicted sccres, 
whereas a discriminant model actually classifies cases as 
predicted successes ci predicted failures. Eecause cf this 
difference, the caliculaticn of cell probabilities is mere 
complicated for regression mcdels than for discrigzinant 


nodels. 
1. Fegressicn Mcdels 


A regression sodel and the data from wnich it nas 
develcred provide itrfcrmatior on the predicted and actval 
scores fcr e€ach case. In order to classify these caseés.into 
the fcur selecticn outcomes, the cut score on the predictor 
and the score cn tke criterion above which mecrle. dare 


considered to be successful must be known. If the criterion 


is censtructed as a dichotcmcus (success/fail) variable, 
then the cases assigned a value of "one" are corsidered 
successful and those with a value of "zero" are corsicered 
unsuccessful. If the criterion 1S a continuous variatle 


(such as length cf service) then a value on the scale mrst 


Le chcsen as the dividing line between success and failure. 


oO 
dad 


The cheice cf the cut Score is net such a Sitple matter eames 


cannct te arbitrarily assigned aS can the distinction 


tetween success and fail. The choice of the cut sccre, as 
menticned Lrefore, cften dererds on the desired selection 
ratic. In the abserce of infcrmation on tre desired seiéc- 


tion ratio, cell frckabilities are calculated for ach o£ 
many fpossitle cut sccres, and a cut score 1s eventually 
choser Lased on which set cf cell probabilities maxinizes 
the utility of the mcdel. In a data set containing deécuae 
and predicted sccres, different sets of ceil protatilitieés 
can ke calculated if each predicted score is considered to 
ke a cut sccre. Tatle XVIII ccntains five fairs of actual 
and predicted scores shich will be used to illustrate tne 
nethcd. 


TABLE XVIIi 
Illustrative Actual and Predicted Scores 


Actual Criterion Predicted Criterion 


SCOre Score 
5.0 44 
ay 46 
ys 4] 
4 € 49 
4s 30 


In this illustration, cases who serve 48 acntks or 
lengexr are considered to Ee successful. Each diferent 
predicted score will cre considered as a cut score and cell 
counts fcr each cut sccre will be calculated. If the cut 
score is 44 months, tken all cases with a fredicted sccre of 
44 mcntks cr mcre sili be accepted, and those with a 


Predicted score of less than 44 months will be rejected. In 


this exaarle, for a cut sccre of 44, all cases will be 
acce;ted. No cne is rejected, therefore, valid negatives 
and faise negatives will ke zero. Of the five cases 
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acce;rted, Paree NN haverdcLudiwmies OL YS menths cr umucre 
(SUCCESSES). Therefcre, the number of valid positives is 
three. Two of the five cases accepted had actual LCS cof 
Jess than 4& months (failures). Therefore, false fesitives 
will te two. Thas the teest set of cell proktabilities that 
result wken the cut score is 44 ares PVP = 3/5, EFF = 2/5, 
FFN = Q0 and PVN = 0. The next set of cell protalilities 
will result when 4€ months is considered to (fe the cut 
SCOre. Cne case hada predicted LOS of less than 46, 
Therefore, he would ke rejected. Has “aGitwead LOS as 50 
tmontks, sc he was falisely rejected, i.e. FN = 1. Nc cne 
else was rejected so VN = 0. Four cases had a peedictec 10S 
cf 4€ cr greater so all four would be accepted. Ct these 
four, txo tad actual LOS of less than 48 months (FE), and 
two had actual LCS of 48 months or more ({VP). Thus fora 
cut sccre cf 46, PVP = 2y5, PFP = 2/5, PFN = 1/5 and EFVN = 
C. This pfrocess is repeated until five sets of cell frecb- 
abilities (one for each different predicted sccre) are 


calculated. 
2. Liscriminant todels 


In a discriminant mcdel the criterion is a categcr- 
ical (0,1) variable. The Output from the SAS Discriminant 
Frocedtre iS a twe by two tabdle where the cases are 
Fredicted to be either a "zero" or a "one", and the 
Fredicticn is compared to the actual score. Table XIX cives 
an atkreviated example of the output from the discrigzinant 
FLOCECUre. 

The columns are the model's predicted scores fcr the 
750 cases in this Eypothetical sampile. Here the gocel 
predicts that 300 of the casés will sccre "zero" cn the 
critericn, and it predicts that 450 of the cases will sccre 
a "one" cn the criterion. The rows are the actual sccres of 


the cases. 250 people actually scored "zero" (failures) and 


TALE 2x 
Illustrative Discriminant Example 


Fredicted 
0 1 Tot al 
100 150 250 
Actual 
200 300 500 
Total 300 450 15a 
500 people actually scored “one" (successes). Because, in 
effect, tke discriginant frecedure chooses its cwno cut 


score, the four cell probabilities can be derived directa 
from the cutfut. Tre predicted "ones" are feople tuat the 
model classifies as accept. Of these 450, 150 actually 
failed sc they are false positives, and the remaining 00 
were successful, so they are valid positives. Of the 3200 
cases tkat the mcdel wculd have rejected (predicted 
"Zercs"), 100 were failures (valid negatives) and z00 were 
successes (false negatives). Again the cell protarilities 
are fecund tLy dividircg each ccunt by the nurber cf casés. 
Therefore, FVP = 30077150, PFP = 1507750, PFN = 2007/3Ceana 
EVN = 17CC,750. For a discritinant model, there is cnly cne 
set cf cell frobablities to bre calculated. 


Fe ESJTIZATION OF CELI UTILITIES 


In crder tc calcrlate the overall utility 9f a mcdel, 
utilities associated sith each selection outcome need to be 
estimated. "Althougk the assignment of utility values to 
cutcches may very well be the ‘Achilles Heelt of cGeéecision 
theory, it is not a probiem that can be ignored [yj any 
institution that makes perscnnél decisions." [Ref. 19] 

Ideally each selection outcome should have a unisguely 


estinated utility. EFecause of the difficulty in estimating 


Demi trecmrcomedch  Cltkcomem (particularily for the false and 


valid necatives), relative utilities are estimated. Lites S 
eepaclent that a  person@ho 15 correctly selected {valid 
positive) has a positive worth to the organizaticn. A 


reascnatle estimate cf this werth is the marginal freduct of 
the eazplcyee. In this study it is assumed that the Lavy 
compensates sailors at the full value of their marginal 
Froduct, and the Billet Cost Model provides an estimate of 
the ccst to the Navy of staffing a billet [Ref. 1€}. 
Eecause relative utilities are the issue at this time, the 
Hemlaty Gf a valid fcesitive (U1) is assigned the value of 
falls 

It is a reasonaktle assumption that the utility cf a 
false positive is a regative number. As the emplcyee was 
not judced to be successful, his marginal product was jfreb- 
ably less than the marginal cost to keep him in the jck. In 
additicn a soor ferfcrmzer may adversely affect the ferforn- 
ance and freductivity of his fellow employees, and waen he 
leaves, additional e€xpense is necessary to find a reflace- 
nent. Cn the other hand, Howtomuniakely that a cor 
perfcrmer dces net centribute anything to the organizaticno, 
and tkus it is cbviously difficult to estimate the magnitude 
cf the disutility of a false positive. Toilet SS Gy oa 
Minor fcrm cf sensitivity analysis 1s undertaken to circun- 
vent this estimation difficulty, and expected overall utili- 
ties are calculated for three different relative values of 
false fositive utility (U2). These vaiues are -.5, -1, and 
a relatively extreme assumpticn, -2. 

The disutility of a false negative is aiso difficult to 
estimate, partly becatse it is nct known what happens tc the 
applicant after he is rejected. If the Navy rejects an 
applicant tc the AD rating but accepts him in another rating 
where te is subseguertly successful, then his disutility 


could ke reasonatly argued to tLe zero. Ii, however, the 
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Navy rcetects him altogether when he would have been 
successful if selected, then the costs of attracting and 
testing hin are wasted and additional costs are reguired to 
attract and test anotter applicant. These costs will depend 
cn the state of the Lrecruiting market at the time. If there 
are many gcod guality applicants then the disutility of 
rejecting a potentially successful applicant may be small. 
Again, as a tyre of sensitivity analysis, three relative 
values fcr the utility of a false negative (U5) are ccnsid- 
ered; 0, =.25 and = .-. 

It is nct obvious that any utility should be assigned to 
U4, the utility cf a valid negative. The person would have 
failed anyway, so nothing was gained by rejecting hin. 
However, wken viewed £rom an €conomist's viewpoint in rela- 
tion tcS CErort unit yeeosrs, the fact that the jperscn was 
correctly rejected means that the organization did nct have 
to kear tke cost of incorrectly accepting someone who turns 
cut tc ke unsuccessful. Thus, correctly rejecting an afrli- 
cant is cf egual and offosite utility to inccrrectly 
accepting him. Therefore, 04 = -U2. 

Tke use of relative utilities iS a conventicr to 
simplify the estimaticn of cell utilities. In the akcve 
discussicn relative utilities are estimated on the tasis 
that tke utility of a valid fesitive is +1. However, the 
values of U1 tarougkh U4 that are used in the f£c0rnuwlaion 
cverall expected utility, (Equations. ie need tc be 
expressed in actual dcliars. As mentioned above, the Eillet 
Cost Model is used tc estimate the utility of a valid fosi- 
tive. The standard manyear cost of an E-4 Aviation 
Fachinist's Mate is $24,163. This cost comes frog finarcial 
year 1983 data and rejresents the total cost to the Navy of 
creating and filling a jok slot over one full year. 
[Ref. 16} A utility of +1 is therefore eguivalert to 
#$24,163, a utility cf -.5 will be -$ 172708727" 400) -ce- 
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ComerECGFAM= USED TO CALCULATE ULILITIES 


Ae mexrtioned in Section Aw. above, the calculaticn of 
See plckdkrilicies icr a £egression model 1s a fairly 
tedicus and repetitive procedure. This section ccrtains 
three saazlie programs used to calculate the expected utility 
c£ a medel. Explaratory comments are provided fcllicwing 
each set cf SAS statements. The first program (Table XX) 
computes the predicted criterion score for each case and 
writes tke results cut to a file called “RTYHATA". Jakle 
XXI shcws part of tke output from the first progranr. The 
second fregram's main purpose (Table XXII) is to calculate 
the cell prebabilities that weuld result if each different 
predicted score were used as a cut score. The céll jfrecb- 
abilities are writter out tc a file called "RTUTILA". The 
Frogran alsc calculates the expected utilities for one set 
cf cell utilities and outputs the 30 largest utilities that 
result (Takle XXIII). The third frogram (Table XXIV) calcu- 


dates tte utilities for six dititferent sets c£ cell 
utilities. 
As e€xplained befcre, only one set of prokatilities 


results fren a discriminant mcdel and these can te readily 
gained frcm the discriminant output. No programs were used 
to calculate the expected utilities of a disScrinminant nodel 


and these calculatiors were dcne by hand. 


[T. CAICCLATION CF BASE LINE UTILITIES 


As described in Chapter V, the utility of the Navy's 
criginal se€lecticn strategy (the base line utilit y) needs to 
ke calculated in crdec for comparisons to ke made. 
Cbservation 4 in Tabie XXIII demonstrates that when all the 
cases are accepted (41.0774 is the lowest predicted sccre), 
the séelecticn ratio is obviously 1 and PVP = .86062€& (which 
4s the tase rate) ard PFP = 1 - PVP = .1393€2. No one is 


SS 


rejectec, therefore PFN and EVN are zero. The expected 


utility under these circumstances is: 
EG = .8€0€28($24, 162) + .159362(-312,0G2) * Cle e0 — ee 


As Table XXIII shows, the paximum utility ecccurs waen 
the cut sccre is slightly higher than the lewest predicted 
score {there are five cases with a predicted score cf less 
than 43.26SZeen Table .2)-2 This maximum utility (2 1S 
is .12 percent greater than the base line utility of 
$197 12. 
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CIUAL AND FREDICTED CRITERION SCCKES: 


-FILEOUT.RIYHATA PLOT; 
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EROC SCK1 LATA=DATA1 CUI=THIRD; 
EY CESCEADING UTIIC: 
LATA IHIEC; 
SEI TEIEC; 
KEEP CSCCRE EVP PFE PFN FYN SRATIO SUCCRATE UIIIC; 
IF NILE 30: 
EROC EFIN1; 
1ITLE EASE UTILITY IS 14061 ANZ BASE SUCCESS RATE IS .8606E; 
Ze 
EROC EICT LATA=TAIRI: 
PLCI CILIIC * SRATIC = '+* ~ VREF =14061; 
EROC SORT LATA=DATA1 CUT=FOURTH: 
EY CESCENDING UTIIL; 
LATA FCUETE: 
SEI FCURTH: 
KEEP CSCCRE PVP PEE EFN EVN SRATIO SUCCRATE UTILD; 
Lt No LE wOe 
EROC EFI¥T; 
TITLE EASE UTILITY IS 19112 AND BASE SUCCESS RATE IS .86€0€; 
FS 
TITLE= j4= 1 = -.5 gf Us= -.5 o U4= 5 “3 
EROC EIC1 LATA=FCURTE: 
PLCT CIIID * SRATIC = '+* ~ VREF =19112; 
EROC SOK1 LATA=DATA1 CUT=FIFTH; 
EY CESCENDING UTIIE; 
[ATA FIFE: 
SEI FIFE: 
KEEP CSCCAE PvP PFE PFN EVN SRATIO SUCCRATE UTILE; 
EROC ERIN: 
TITLE EASE UTILIIY IS 17428 ANZ BASE SUCCESS RATE IS .8606 
Zs 
TITLE= Ul= 1 U2= =1 , Us= -.5 , U4= 1 .: 
EROC ELC LATA= FirT a: 
FICT CIILE * SRATIC = '+" ~ VREF =17428; 
EFROC SCKI LATA=DATA1 CUI=SIRTH: 
EY LESCENDING UTIIF; 
LATA SIXTE; 
SE1 SIXtH: 
KEEFE CSCCKE FVP PFE FFN EVN SRATIO SUCCRATE UTILF; 
IF NILE 30; 
FROG ERIRI: 
LITLE EASE UTILITY IS 14061 AND EASE SUCCESS RATE IS .8€0€; 
a 
TITLE? Ul= 1 , UZ= -2 , U3= =.5 2 d= 2 -3 
FROC ELCI LATA=SiXTH: 
PLCT UIIIF * SRATIC = '+* /~ VREF =14061; 
/* 
// 
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